Skip to main content

Backup Exivity Installation on Kubernetes using Velero

This document provides guidance on backing up an Exivity installation deployed on Kubernetes using Velero. Effective backups ensure operational continuity by safeguarding all critical data associated with Exivity.

Third-party backup tooling

Velero, object storage, snapshot controllers, PostgreSQL backup tools, and storage replication platforms are third-party products. Exivity documents what application data must be protected, but you are responsible for implementing, operating, testing, securing, and supporting your backup tooling.

Overview​

The Exivity installation on Kubernetes includes three critical components to back up:

  1. 🐇 RabbitMQ: Can be deployed via the Helm chart or configured as an external service. RabbitMQ is not the primary system of record, but its availability still affects running work.
  2. 🐘 PostgreSQL: Can be deployed via the Helm chart or configured as an external service (recommended for production; see Helm Chart Configuration & Dependencies). PostgreSQL is the primary system of record and should have database-native backups.
  3. 💾 Filesystem: Persistent Volume Claims (PVCs) associated with the Exivity namespace.

Note: If RabbitMQ and PostgreSQL are configured as external services, you must establish additional backup procedures for these components. This document does not cover external service backups.

The Helm chart also includes optional PostgreSQL backup and restore jobs under service.backup and service.restore. These jobs are intended for chart-managed database backup workflows and are separate from Velero namespace/PVC backups.

Prerequisites​

  1. Velero Installation: Ensure Velero is installed on your cluster. Refer to the Velero file system backup documentation for detailed setup instructions.
  2. Backup Storage Location: Configure a Backup Storage Location (BSL) to store backups, such as AWS S3, Azure Blob Storage, or NFS.
  3. Cluster Access: Ensure you have kubectl access to the Kubernetes cluster with the necessary permissions to manage backups.
  4. Namespace Details: Identify the namespace where Exivity is deployed (e.g., exivity or a custom namespace).

Filesystem Backup (PVCs)​

Persistent Volume Claims (PVCs) in the Exivity namespace store critical application data. Backups should prioritize PVCs containing application data, while PVCs ending with -log, which store log files, are less critical but still recommended to back up.

Important PVCs to Back Up​

All PVCs not ending in -log should be backed up as they contain essential application data.

Log PVCs​

PVCs ending with -log store log files. These are not critical for application recovery but are still recommended for backup to retain logging history.

Backup Methods with Velero​

Velero supports two primary methods for backing up data:

1. Snapshots​

Snapshots offer a quick and efficient way to create point-in-time backups of PVCs. However, this method depends on the cloud provider's support for CSI snapshots.

Note: Self-managed clusters may not have snapshot capabilities. In such cases, consider using Restic as an alternative.

2. File System Backup​

Velero file system backup provides a flexible approach, directly copying PVC data to the backup storage location. It is particularly useful when snapshots are unavailable.

For recent Velero versions, enable node-agent based file system backups during installation. Check the Velero documentation for the exact flags for your version.

For older Velero versions, this feature was referred to as Restic integration.

For more details, visit the Velero file system backup documentation.

Helm Chart Database Backup Jobs​

The chart can create a PostgreSQL backup PVC and run backup or restore jobs when enabled:

service:
  backup:
    enabled: true
    retentionDays: 7
    filePrefix: "backup"
    storage:
      size: "10Gi"
      storageClass: ""
  restore:
    enabled: false

Do not enable service.backup.enabled and service.restore.enabled at the same time. The chart will fail rendering if both are enabled.

The chart restore job is primarily designed for the embedded PostgreSQL database. For managed PostgreSQL services, use the provider's native backup and restore tools.

Verifying and Validating Backups​

Step 1: Verify the Backup​

Ensure the backup completed successfully:

velero backup describe exivity-backup --details

Step 2: Validate the Backup​

Simulate a restore in a non-production environment to verify the integrity of the backup.

Restoring from Backup​

Step 1: Restore the Namespace​

To restore the Exivity namespace and associated PVCs:

velero restore create --from-backup exivity-backup

Step 2: Verify the Restore​

Check the restore status:

velero restore describe <restore-name>

Ensure all Pods are running:

kubectl get pods -n <exivity-namespace>

Additional Considerations​

  1. Application Downtime: Minimize activity on PVCs during backups to prevent data corruption. Consider putting Exivity services in maintenance mode.
  2. Backup Scheduling: Automate backups by setting up a CronJob to ensure regular backups.
  3. Disaster Recovery: Store backups in a separate region or location for added resilience.
  4. External Services: For PostgreSQL and RabbitMQ configured as external services, establish a dedicated backup strategy.
  5. Security: Encrypt backups and limit access to backup storage for security.

By following this guide, you can effectively back up and restore your Exivity installation on Kubernetes using Velero, ensuring minimal disruption in case of data loss or system failure.